Claims 



An information input processing, gesture-key mapping 
computer system, the system comprising one or several 
camteras, one or more memories with CPU connected to the 
cameivas, and processes running in the CPU that associates 
gesture movements with typing and produce gesture 
associated textual output. 

2. The gesture -key mapping system as in claim 1, where a 
ieedback is provided to the user on what kind of keys are 

jsociated with the user's gestures. 

3. \The gesture -key mapping system as in claim 2, where 
the feedback is provided using one or more of the 
following: displaying keys on a display, playing sounds 
labels £or keys, displaying image indicators on a 
display, \? laying special sound indicators, projecting the 
keyboard ftp any surface, and displaying picture of the 
keyboard wi^h user ' s hands . 

4. The systemas in claim 1, where the gesture-key 
processing is provided using the following modules: 

a) a gesture capturing module that captures gestures 
through camera sensors ; 

b) a gesture classif iciator module that classifies 
gestures into classes of gesture movements; 
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c) an associator module for associating gesture classes 
or sequence of gesture classes with one or several most 
provable keys; and 

d) An integrator module that integrate sequence of 
cand\date of most probable keys into unique output key 
sequence . 

5. The system as in claim 4, where the integrator module 
includes one or more of the following: 

a) languag^ module component that estimate probabilities 
of word strings corresponding to key candidate sequences; 

b) character frequency module that estimate probabilities 
of character Strings corresponding key candidate 
sequences ; 

c) confusable matrix that estimate how often correct 
gesture classes ^re confusable with another gesture 
classes; 

d) gesture classes \probability model that estimate 
probability of observing a string of gesture classes 
given a sequence of Yjesture frames; 

e) computation of a probability of production a sequence 
of keys given a strinA of gesture frames; 

f) generation of a lattVce of sequences of keys given 
sequence of gesture frames; 
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g) finding the most probable sequence of keys from the 
lattice of\ key candidate strings. 

6. A systern according to Claim 5, wherein each sequence 
of keys receives a probability score, and the sequences 
of keys that\receive low scores are removed and are not 
continuing when new candidates for keys arrive. 

7. A system according to Claim 1, further comprising a 
gesture correlator module to allow to adjust 
automatically aA invisible keyboard to hand positions. 

The method for producing a textual output in which a 
us\r makes typing like gestures without the presence of a 
keyboard and the gestures are associated with the most 
probable keys that would be typed if a keyboard were 
presented. 

9. ^method according to Claim 8, wherein the proability 
is computed using HMM. 

IN). A method of typing using a virtual keyboard, 
comfenrising the steps: 

makinV typing gestures without any real keyboard; 
sensing the typing gestures; and 

producinV from the sensed typing gestures, gesture 
associatecNtextual output. 

11\A method according to Claim 10, wherein the typing 
gestures are made by a person, and further comprising the 
steps\>f providing feedback to the person on texture 
output ^a^socia ted with the gestures. 
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12. A method according to Claim 11, wherein the step of 
providing feedback includes the step of displaying an 
linage of typing keys associated with the gestures. 

13 A A method according to Claim 10, further comprising 
the\step of generating an image of a keyboard; and 
wherein the step of making typing gestures includes the 
step ©f making typing gestures relative to said image os 
if sain image were a real keyboard. 

14. A meVhod according to Claim 10, wherein the producing 
step incudes the step of classifying gestures into 
classes on gesture movement. 

15. A method according to Claim 14, wherein the producing 
step furtheA includes the step of associating gesture 
classes with ^individual typing keys. 

16. A method according to Claim 10, further comprising 
providing train^ig data in words or sentences with 
certain timing dkta. 

A typing system using a virtual keyboard, comprising 
meafas for sensing typing gestures made without any real 
keyboard ; and 

means for producing, from the sensed typing gestures, 
gesture associated textual output. 

lV. A system according to Claim 17, wherein the typing 
gestures are made by a person, and further comprising 
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means fir providing feedback to the person on texture 
output associated with the gestures. 

19. A system according to Claim 18, wherein the means for 
providing feedback includes means for displaying an image 
of typing\keys associated with the gestures. 

20. A system according to Claim 17, further comprising 
means for generating an image of a keyboard; and wherein 
the sensing means includes means for sensing typing 
gestures made\ relative to said image os if said image 
were a real keyboard. 
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21. A system according to Claim 17, wherein the producing 
means includes means for classifying gestures into 
classes of gesture movement. 

22. A system according to Claim 21, wherein the producing 
means further includes means for associating gesture 
classes with individual typing keys. 
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